Method of temporal resampling and apparent motion speed change for motion picture data

ABSTRACT

The present invention is a method of processing a sequence of frames from digital motion picture cameras to allow remapping of the time-base of the input frames to a new and possibly variable time-base in the output frame sequence. This invention provides a method that makes small adjustments to the user&#39;s specified positions to achieve optimal integer mapping from input frames to output frames in the integrated function. The present invention provides for an algorithm for frame rate resampling providing a method for determining the mapping of input frames to output frames and a method for smoothly ramping the mapping of input frames to output frames based on interactive user input and a method for optimizing the alignment of input frames to output frames such that during areas of constant speed the alignment is optimally centered on the frames and a method for computing the weighting function for averaging input frames to output frames and a method for handling negative values in the weighting function in regions of bright highlights.

This application claims the benefit of U.S. Provisional Application No.62/185,592, filed Jun. 27, 2015 and the entire content is incorporatedby reference herein and made part of this specification.

FIELD OF THE INVENTION

The present invention provides a method of processing a sequence offrames from digital motion picture cameras to allow remapping of thetime-base of the input frames to a new and possibly variable time-basein the output frame sequence.

BACKGROUND OF THE INVENTION

The present invention provides for a method of transforming over time(or temporally resampling) a set of input frames from a digital cinemacamera to produce an output sequence of frames with different temporalsampling, meaning different playback frame rate, variable apparentmotion rate, or a combination of both.

Digital cinema cameras are not often capable of “speed ramps,” which arecommonly used with film cameras. In a film camera, the frame rate ofacquisition is changed continuously during a shot, allowing the apparentspeed of motion of actors or objects in the scene to vary duringconstant frame rate playback. The artistic uses of this technique werewidely varied, but the limitations of digital cameras have preventedthis technique.

Current methods of temporal resampling are limited by their ability togenerate a proper transform function and to apply a finite impulseresponse (FIR) filter.

The transform function is the function that provides the mapping betweenthe input frame set and the output frame set and vice versa. It isdesired that the transform function be user variable, but it is alsodesirable that the transformation function map integer frame numbers asoften as possible. These are sometimes contradictory requirements, asallowing the user full control of the resampling function, and allowingthe resampling function to have a smooth derivative will usually resultin non-integer mapping of frame numbers.

Once transformed from input to output frames, there is still the problemof properly resampling the input frames surrounding the time center tocreate an output frame with good temporal response. This involves theuse of a finite impulse response (FIR) filter to perform a weightedaverage of input frames for each output frame. Proper FIR creation oftenresults in weighting with negative coefficients. As a principle ofsignal processing, this is mathematically correct, but in practice thenegative weighting coefficients present a problem when very brighthighlights in the image set are later clipped in editing. The negativevalues created from these very positive input data can present unwelcomeartifacts when the positive data is later clipped. Restricting FIRcoefficients to only positive values dramatically limits the possiblefrequency responses available for resampling.

Current methods of frame rate resampling rely upon simplenearest-neighbor mapping of input to output frame numbers, and in aspeed ramp often result in stuttered or jerky motion. The presentinvention optimizes the temporal relationship between the input andoutput frames, and by adjusting the weighting of the FIR filter,achieves a much more accurate motion look. The smooth FIR waveform, inturn, can result in negative image values, and the damped subtraction inthe present invention alleviates potential discontinuities based onsensor clipping or color adjustment in post production.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The accompanying drawings in the attachment, which are incorporated intoand form a part of the specification, illustrate one or more embodimentsof the present invention and, together with the description, serve toexplain the principles of the invention. The drawings are only for thepurpose of illustrating one or more preferred embodiments of theinvention and are not to be construed as limiting the invention. In thedrawings:

FIG. 1 is a curve illustrating a function that is formed as a piecewisepolynomial;

FIG. 2 is a flowchart depicting an implementation of the method oftemporal resampling according to an embodiment of the present invention;and

FIG. 3 is a flowchart depicting a method for refining the temporalremapping function to optimally map integer input frame numbers tointeger output frame numbers while allowing user input.

SUMMARY

The present invention comprises a method for frame rate resampling, themethod comprising using an algorithm whereby the algorithm provides forsmoothly ramping apparent speed and creating the output frames by usinga plurality of input frames for each output frame. The method comprisessmoothly ramping apparent speed comprising aligning created outputframes with appropriate input frames.

The method of the present invention comprises determining input framescorresponding to each output frame comprising optimizing the piecewisepolynomial and producing an output frame using a minimizing solver tooptimally achieve an integer non-fractional mapping.

The present invention further comprises a method for frame rateresampling, the method further comprising an integer mapping from outputto input frames.

The present invention further comprises a method for frame rateresampling comprising solving for a repeating pattern of fractionalinput frame locations wherein the average absolute fractional offsetfrom an integer number frame is kept constant. The method furthercomprises creating the output frames by weighting the input frames bycoefficients of a finite impulse response (FIR) filter and resolvingstrong negative values resulting from the FIR filter by means of adamped subtraction. The method comprises producing the dampedsubtraction using a function with a slope (derivative) of −1 between 0and 0.2, and monotonically decreasing above 0.2 but never negative.

DETAILED DESCRIPTION OF THE INVENTION

The present invention addresses and improves on deficiencies in devicesand methods currently used. The transformation function from inputframes to output frames is specified initially by the user by indicatingkeyframes with desired transformation speeds at those locations. Currentlimitations poorly handle the inherent non-integer values that suchspecifications will lead to, since the user is specifying the firstderivative of the actual transformation (rate rather than actualposition). This invention comprises a method that makes smalladjustments to the user's specified positions to achieve optimal integermapping from input frames to output frames in the integrated function.Also, while finite impulse response filters have been employed toresample digital cinema frames, the problem of negative values inwell-formed FIR coefficients has not been previously addressed withrespect to very bright highlights in the footage. This inventionaddresses that issue and comprises a method to correct this problem.

The present invention provides for an algorithm for frame rateresampling comprising a method for determining the mapping of inputframes to output frames; a method for smoothly ramping the mapping ofinput frames to output frames based on interactive user input; a methodfor optimizing the alignment of input frames to output frames such thatduring areas of constant speed the alignment is optimally centered onthe frames; a method for computing the weighting function for averaginginput frames to output frames; and a method for handling negative valuesin the weighting function in regions of bright highlights.

Frame mapping comprises the mapping of input frames to output frameswith variable speed and allowing user interactive input starts withspecified user points, or keyframes. Each point in the timeline of theacquired clip from the digital camera is specified by the user, alongwith the desired playback speed at that point.

The algorithm computes a piecewise polynomial spline to form a function“f(if)”, which is the function returning speed for a given input framenumber “if.” The first integral of f(if) is g(if). This function “g(if)”will return the output frame number corresponding to the input framenumber “if.”

The inverse of “g(if)” is “invg(of)”. The function invg(of) will returnthe input frame number corresponding to the output frame number “of.”

A desirable property of the functions g and invg is that for regions ofconstant speed and of integer resampling rates, the mapping is frominteger to integer numbers. That is, if “if” is an integer, it isdesired that g(if) also be an integer value with no fractional part. Inthe case of the entire clip being of constant speed, this is trivial.However, if an earlier region of the clip has undergone a speed ramp, itis important to adjust the piecewise polynomial.

The present invention comprises a method of adjustment and is describedherein. For each region between user-specified keyframes, if the regionis of constant speed (i.e. both endpoints have the same speed value), aminimizing solver is employed to apply an adjustment (between −0.5frames and 0.5 frames) to the position of the “start” endpoint of theregion until mapping in the region is optimally integer-to-integer.

Damped subtraction is used to produce an output frame from a pluralityof input frames surrounding the point specified by the “invg” function,wherein the input frames are weighted (or multiplied) by an array ofvalues and then summed to produce a single output frame. The values forweighting during this downsampling operation are called the “resamplingkernel” or “synthetic shutter.”

Good resampling kernels will often have negative values. Therefore someframes from the input footage are effectively subtracted from theoutput. A method for smoothly limiting the amount of subtraction of verybright regions in the image is desired, because the input frames have avery wide dynamic range that is later trimmed during editing. Such amethod is to limit the subtraction based on the overall amount ofsubtraction allowed. This is accomplished by first accumulating the sumof all frames scaled by positive values of the resampling kernel andnaming this image “m.” Similarly, all negative values of the resamplingkernel are made positive (complemented) and used to scale and sum theappropriate frames into a single image named “s.” The final output imagewould normally be computed by the simple relationship: output=m−s. Ifthe expression “output=m−s” is replaced by the expression“output=m*h(s/m)”, it is equivalent as long as the function h(x) isdefined as “h(x)=1−x.”

For damped subtraction, however, the expression h(x) is replaced with anew function, d(x), which has several damping properties, i.e: d(x) isthe same as h(x) in the region between x=−Inf and x=0.2; d(x) iscontinuous; d(x) is monotonically decreasing; d(x) is always positive;and d(x) at x=0.2 has a continuous derivative equal to −1.

Such a function can be formed as a piecewise polynomial having thefollowing shape, as seen in FIG. 1.

FIG. 2 is a flowchart depicting an implementation of the method oftemporal resampling according to an embodiment of the present invention;and

FIG. 3 is a flowchart depicting a method for refining the temporalremapping function to optimally map integer input frame numbers tointeger output frame numbers while allowing user input.

Although the invention has been described in detail with particularreference to these preferred embodiments, other embodiments can achievethe same results. Variations and modifications of the present inventionare obvious to those skilled in the art and it is intended to cover allsuch modifications and equivalents.

What is claimed is:
 1. A method for frame rate resampling, the methodcomprising: using an algorithm whereby the algorithm provides forsmoothly ramping apparent speed; and creating the output frames by usinga plurality of input frames for each output frame.
 2. The method ofclaim 1 whereby smoothly ramping apparent speed comprises aligningcreated output frames with appropriate input frames;
 3. The method ofclaim 2 wherein determining input frames corresponding to each outputframe comprises optimizing the piecewise polynomial and producing anoutput frame using a minimizing solver to optimally achieve an integernon-fractional mapping;
 4. A method for frame rate resampling, themethod further comprising an integer mapping from output to inputframes;
 5. A method for frame rate resampling comprising solving for arepeating pattern of fractional input frame locations where the averageabsolute fractional offset from an integer number frame is keptconstant;
 6. The method of claim 5 further comprising creating theoutput frames by weighting the input frames by coefficients of a finiteimpulse response (FIR) filter and resolving strong negative valuesresulting from the FIR filter by producing a damped subtraction; and 7.The method of claim 6 wherein producing the damped subtraction using afunction with a slope (derivative) of −1 between 0 and 0.2, andmonotonically decreasing above 0.2 but never negative.